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Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 
* Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 

Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )S Responsive to communication(s) filed on 14 January 2004 . 
2a)D This action is FINAL. 2b)M This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quay/e, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) S Claim(s) 1-13 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) I3 Claim(s) 1-13 is/are rejected. 

7) \3 Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10) 13 The drawing(s) filed on 18 November 2003 is/are: a)E3 accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1121(d). 

1 1) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12) 13 Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 
a)S All b)D Some * c)D None of: 

1 .M Certified copies of the priority documents have been received. 

2. Q Certified copies of the priority documents have been received in Application No. . 

3. Q Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 
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DETAILED ACTION 
Continued Examination Under 37 CFR 1.114 

1 . A request for continued examination under 37 CFR 1.114, including the fee set forth in 
37 CFR 1 .17(e), was filed in this application after final rejection. Since this application is 
eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1 .17(e) 
has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 
37 CFR 1.114. Applicant's submission filed on January 14, 2004 has been entered. 

Drawings 

2. The drawings were received on November 18, 2003. These drawings are acceptable. 

Response to Arguments 

3. Applicant's arguments on page 8, first paragraph, filed November 18, 2003, with respect 
to claims 5-7 have been fully considered and are persuasive. The 35 U.S.C. § 1 12 1 st paragraph 
rejection of claims 5-7 has been withdrawn. The objections to claims 1 and 2 for informalities 
and claims 9 and 12 for being substantial duplicates have also been withdrawn in view of 
applicant's amendments to those claims. 

4. Applicant's arguments on page 9, third paragraph, filed January 14, 2004, with respect to 
Suzuki have been fully considered but they are not persuasive. Applicant states, "Although the 
CCD cameras of Suzuki move about an axis (i.e. rotate) their position does not change." The 
examiner respectfully disagrees. First of all, Suzuki does not disclose that the CCD cameras 
rotate or move about an axis. While it is not explicitly discussed, Suzuki does indeed disclose 
that the CCD camera pair may be moved around sample 10 or the sample may be rotated in order 
to acquire images of the sample from various directions (column 3, line 66-column 4, line 9). 
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Additionally, newly cited reference USPN 4,187,051 to Kirsch et al. has been applied to claims 
9-12 to show another example of placing a camera (image pickup device) on a movable and 
positionable part of a robot. 

5. Applicant's arguments with respect to claims 1, 2, 8, 9, 12, and 13 have been considered 
but are moot in view of the new ground(s) of rejection. 

Claim Objections 

6. Claim 1 is objected to because of the following informalities: line 1 1 has been amended 
to recite, "said image pickup device is fixed to a movable and positionable part of the robot" but 
this is the first occurrence of the word "robot." It is believed that applicant did not intend to 
change "part of a robot" to "part of the robot" (emphasis added). Appropriate correction is 
required. 

7. Claim 2 is objected to because of the following informalities: line 13 recites the word 
"gasped" where it is believed applicant intended to write the word "grasped " Appropriate 
correction is required. 

8. Claim 13 is objected to because of the following informalities: line 5-7 states, "taking a 
current image of a workpiece shaped like the subject; and then before picking up the subject 
determining a current workpiece-camera orientation. . ." It is believed that applicant intended this 
to read, "before picking up the workpiece". Appropriate correction is required. 

Claim Rejections - 35 USC § 102 

9. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 
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(b) the invention was patented or described in a printed publication in this or a foreign country or in public use or on 
sale in this country, more than one year prior to the date of application for patent in the United States. 



10. Claim 13 is rejected under 35 U.S.C. 102(b) as being anticipated by Herndon et al. 
(USPN 4,909,376, newly cited, "Herndon"). 

1 1 . Regarding claim 13, Herndon discloses a method comprising: robotically taking images 
of a subject (physical part/component) with different subject-camera arrangements (column 6, 
line 29-49), and associating with each image or data thereof information indicating its subject- 
camera arrangement (column 6, line 50-column 7, line 14); then taking a current image of a 
workpiece (component) shaped like the subject (column 3, line 58-column 4, line 6); and then 
before picking up the subject (workpiece, component) determining a current workpiece-camera 
orientation by matching one of the images or data thereof with the current image (column 4, line 
7-20), and using predetermined subject-camera arrangement information of the matched image to 
determine the orientation of the workpiece relative to the camera (column 4, line 32-42). 



12. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

13. Claims 1,3,4, and 8 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Herndon in view of Suzuki (USPN 4,835,450, previously cited). 

14. Regarding claims 1 and 8, Hemdon discloses a teaching model generating method and 
device for image processing, in which a subject object has the same or substantially similar shape 
as that of a reference object, the device comprising: an image processing system with which 



Claim Rejections - 35 USC § 103 
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current three-dimensional orientation of the subject object relative to an image pickup device is 
recognized based on a plurality of predetermined teaching models (component database) of the 
reference object (column 4, line 12-31); and an image-capture system, in advance of the 
recognizing, generating and storing the plurality of teaching models on the basis of respective 
image data produced by taking images of said reference object from a plurality of directions 
(column 6, line 29-49). 

15. Herndon does not disclose that one of the reference object and said image pickup device 
is fixed to a movable and positionable part of the robot or is grasped with a hand of the robot or 
that said robot is operated for positioning to a plurality of different image pickup positions and 
directions. Suzuki discloses a method and system for controlling a robot wherein images of a 
sample are acquired by moving a pair of CCD cameras around the sample, or rotating the sample 
in front of the cameras, and said robot is operated for positioning to a plurality of different image 
pickup positions and directions (column 3, line 66 to column 4, line 9), so that the image data 
respectively obtained at each of said different image pickup positions and direction information 
indicating the respective different direction is stored as a teaching model (column 4, line 32-39). 

16. Regarding claim 3, Herndon discloses that said teaching model is a part of the image data 
of the reference object (column 6, line 29-35). 

17. Regarding claim 4, Herndon discloses that said teaching model comprises data obtained 
by performing image processing on the image data of the reference object (column 6, line 36- 
49). 
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18. Claims 2, 5, and 6 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Herndon in view of Suzuki in view of Werth et al. (USPN 4,504,970, previously cited, "Werth"). 

19. Regarding claim 2, Herndon discloses a teaching model generating device for image 
processing, in which a subject object has same or substantially similar shape as that of a 
reference object, the device comprising: an image processing system with which a current three- 
dimensional orientation of the subject object relative to an image pickup device is recognized by 
carrying out pattern matching processing of an image of the subject based on a plurality of pre- 
determined teaching models of the reference object; and an image capture system, in advance of 
the recognizing, generating and storing the plurality of teaching models on the basis of respective 
image data produced by taking images of said reference object from a plurality of directions, 
wherein said image pickup device is fixed to a movable and positionable part of a robot, which is 
operated for positioning to a plurality of different relative image pickup positions and directions 
(disclosed by Suzuki), so that the image data respectively obtained at each of said different 
image pickup positions is stored as a teaching model (see above discussion of claim 1). 

20. Neither Herndon nor Suzuki discloses that the reference object is fixed to a movable part 
of a first robot. Werth discloses a training controller for pattern processing system wherein it is 
suggested that an application could utilize two robot arms, one which holds a camera which 
visually guides it to observe a precise assembly point and a second which brings a tool or 
assembly within the visual field of the camera where it is visually guided through an operation 
(column 5, line 12-17). It would have been obvious to one of ordinary skill in the art at the time 
the invention was made to utilize two robot arms, one to hold a camera and one to hold a tool or 
assembly as taught by Werth in order to provide more degrees of freedom allowing more views 
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of the workpiece from different directions and to provide proper alignment for mating parts in 
automated assembly operations or move a tool to a specific point on the part (column 5, line 6- 
11). 

21 . Regarding claim 5, Suzuki discloses that said teaching model is generated for every 
direction in which said image pickup device took the image of said reference object (column 4, 
line 2-9) and said teaching model is stored in association with the information on the direction 
(column 4, line 32-39). 

22. Regarding claim 6, Herndon discloses that said image pickup device is a camera (element 
126; column 3, line 58-68). 



23. Claim 7 is rejected under 35 U.S.C 103(a) as being unpatentable over Herndon in view 
of Suzuki in view of Werth as applied to claim 2 above, and further in view of Ninomiya et al. 
(USPN 4,611,292, previously cited, "Ninomiya"). 

24. Herndon does not disclose that said image pickup device is a three-dimensional visual 
sensor that measures a distance between the image pickup means and a plurality of points on the 
object. Ninomiya discloses a robot vision system including a three-dimensional visual sensor 
whose image pickup means measures the distance between the image pickup means and a 
plurality of points on the object (column 4, line 28-49). It would have been obvious to one of 
ordinary skill in the art at the time the invention was made to utilize a three-dimensional visual 
sensor as taught by Ninomiya in order to determine the position and posture of an object without 
operation or accuracy being effected by contrast, color, or surface condition of the object 
(column 10, line 35-39). 
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25. Claims 9-12 are rejected under 35 U.S.C. 103(a) as being unpatentable over Herndon in 
view of Kirsch et al. (USPN 4,187,051, newly cited, "Kirsch"). 

26. Regarding claim 9, Herndon discloses a method of automatic orientation recognition, 
comprising: generating and storing a set of images of different relative orientations of a subject 
(column 6, line 29-49), the images having been captured by a plurality of robotic operations 
corresponding to the different relative orientations of the subject, and associating with each 
image information indicating its respective relative orientation of the subject (column 6, line 50- 
column 7, line 14); after the generating and storing, from a known current orientation of a robot, 
capturing a current image of a workpiece with an unknown orientation relative to an image 
pickup device before the robot has come into contact with the workpiece, where the workpiece 
has a shape substantially similar to the shape of the subject (column 3, line 58-column 4, line 6); 
after the capturing, using pattern matching (comparison) to match one of the stored images 
(database) with the current image (column 4, line 12-3 1); and after the pattern matching, and 
before the robot has come into contact with the workpiece, determining the orientation of the 
workpiece relative to the image pickup device based on the relative orientation information 
associated with the matched stored image, and also based on the known current orientation of the 
robot (column 4, line 32-42). 

27. Herndon does not disclose that a current image of a workpiece is captured with a known 
current arrangement of an image pickup device on a robot. Kirsch discloses a device for video 
article centering, orienting and transfer wherein a rotatable horizontal video camera 22, 86 is 
mounted on robot 24 (Figure 1; column 2, line 54-65; column 4, line 12-25; column 11, line 3- 
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46). It would have been obvious to one of ordinary skill in the art at the time the invention was 
made to mount an image pickup device on a robot as taught by Kirsch in order to eliminate 
complex trigonometric computation by moving the camera with the robot to match a standard 
comparison image of a workpiece properly positioned, centered, and oriented image (column 7, 
line 34-40) in order to determine the precise location, centration, and orientation of the 
workpiece (column 1, line 34-39). 

28. Regarding claim 10, Herndon discloses automatically maneuvering the robot to the 
workpiece (component) based at least on the determined orientation of the workpiece relative to 
the robot (column 4, line 12-20). 

29. Regarding claim 11, Herndon discloses that the generating and storing is performed for a 
plurality of differently shaped subjects (column 6, line 29-35; column 7, line 27-42), wherein the 
current image includes a plurality of differently shaped workpieces, and wherein the pattern 
matching further comprises identifying workpieces from among the plurality of differently 
shaped workpieces using the images and orientation information of the plurality of differently 
shaped subjects (column 8, line 23-42). 

30. Regarding claim 12, claim 9 recites at least all the limitations recited in claim 12 wherein 
the word "arrangement" is used in place of the word "orientation." Note that claim 12 does not 
contain a limitation of determining the arrangement of the workpiece "before the robot has come 
into contact with the workpiece." 

Conclusion 

3 1 . The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. 
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o GB 2,123,172 to Archibald et al. discloses a robot control system having a camera 

mounted on a robot arm to obtain images for comparison with stored images, 
o JP 02-1 10788 to Koiso et al discloses a method for recognizing the shape, position, 

and posture of a three-dimensional object, 
o USPN 5,446,835 to Iida et al (corresponding to JP 07-3 19525 cited on applicant's 

IDS) discloses a high-speed picking system for stacked parts including a storage unit 

that stores a plurality of collative models, 
o USPN 5,621,807 to Eibert et al. discloses an intelligent range image camera for object 

measurement wherein 3-D object geometry data is acquired and compared with stored 

models to identify an object as well as its position and orientation or attitude, 
o USPN 5,790,687 to McLaughlin et al. discloses a method and apparatus for optical 

determination of the orientation of a garment workpiece. 
o USPN 5,987,591 to Jyumonji discloses a multiple-sensor robot system for obtaining 

two-dimensional image and three-dimensional position information. 
o USPN 6,349,245 to Finlay discloses a method and apparatus for registration of a 

robot including an image acquiring arrangement carried by the robot, 
o USPN 6,490,369 to Beiman discloses a method of viewing and identifying a part for a 

robot manipulator including recognizing the location and orientation of the part. 
Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Ryan J Hesseltine whose telephone number is 703-306-4069. 
The examiner can normally be reached on Monday - Friday, 8:30 AM - 5 PM. 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Amelia Au can be reached on 703-308-6604. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 
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April 29, 2004 



